IMAGE PROCESSING DEVICE, PROGRAM PRODUCT AND SYSTEM 



[0001] This application is based on the Japanese Patent 
Application No. 2000-195146 filed on June 28, 2000, the 
contents of which are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0002] The present invention relates to a device, a program 
product and a system for image processing. In particular, 
the invention relates to a device, a program product and a 
system for executing a process to judge whether character 
images in input image data should be converted into character 
code data. 

2. Description of the Related Art 

[0003] Latest image recognition devices can recognize 
character images as character codes with extremely high 
accuracies as long as the documents are of good scanning 
conditions (for example if a document is made up of a single 
font type). However, if the quality of characters on the 
document is bad, or if the layout of characters or the like 
on the document is complicated, the accuracy of recognizing 
character images as character codes drops substantially and 
character image recognition errors occur more frequently. 



[0004] In order to cope with such a problem , systems have 
been known wherein the entire image data obtained by scanning 
the document is stored as a back up* However, the volume of 
a file containing the backup image data naturally becomes 
substantially larger than the input image data, 
[ 0005 ] Therefore , other systems have been proposed wherein 
only the character images that have high probability of 
recognition errors are outputted as character image data (e.g., 
bitmap type image data ) without converting them into character 
code data. The probability of recognition errors is judged 
from information on certainty of recognizing character codes 
measured by checking similarity with the prerecorded standard 
character pattern, etc. 

[0006] However, the information on certainty of 

recognizing character codes alone does not provide a 
sufficient accuracy for a judgment on which character images 
in the image data are accurately recognized as character codes 
and which character images are mistakenly recognized, so that 
it is difficult to remove mistakenly recognized character 
images completely. 

[0007] The types of documents to be scanned have become 
quite diversified in recent years, many of them being color 
documents or documents with complex layouts. As a result, 
it has become increasingly difficult to extract from image 



data character areas where character images exist . Therefore , 
scanning of a document such as the one shown in Fig. 1, where 
non-character graphics are embedded in character areas, may 
result in outputting character code data as a result of 
mistakenly recognizing character codes from graphic images, 
i.e., non-character images in the obtained image data, as 
shown in Fig. 2. The symbol El in the drawing represents 
character code data obtained from character codes mistakenly 
recognized from graphic images, and the symbol E2 represents 
character code data obtained from character codes mistakenly 
recognized from character images. Moreover, there are cases 
where the certainty of recognizing character codes in 
converting graphic images to character code data El is not 
much different from the same for true character images . 
Therefore, it is impossible to remove the character code data 
obtained by mistakenly recognizing graphic images, if a 
judgment for converting into character code data is made only 
on the information of certainty of recognizing character 
codes . 

[0008] Unexamined Publication No. JP-A-8-55185 proposed 
a technology of extracting character areas by means of checking 
positional relation with neighboring character images in the 
stage of extracting character areas where character images 
exist in the image data. USP 5,949,906 proposed a technology 



of first extracting character candidate areas which can be 
candidates for character areas by checking positional 
relation with neighboring character images, and then 
reconstruct character areas based on character images for 
which character codes can be recognized among image data in 
the character candidate areas. However, all of these 
technologies are nothing but the technologies to extract 
character areas from the image data. 

SUMMARY OF THE INVENTION 
[0009] It is an object of the present invention to provide 
an improved image processing device, a program product and 
a system to solve the above problems. 

[0010] It is a further object of the present invention to 
provide an image processing device , a program product and 
a system that are capable of effectively preventing the output 
of mistakenly recognized character code data while increasing 
the ratio of conversion of character images contained in input 
image data to character code data. 

[0011] According to an aspect of the invention, it is an 
image processing device comprising: a character recognition 
unit that recognizes character codes from character images 
in image data; a conversion unit for converting character 
images to character code data according to character codes ; 



and a judgment unit that obtains a degree of character 
continuity, which is a degree of continuity between a character 
image and neighboring character images thereof , for any 
character image for which a character code has been recognized 
by said character recognition unit, and that makes a judgment 
on whether said character image should be converted into 
character code data based on said degree of character 
continuity. 

[0012] According to this invention, it is possible to 
prevent graphic images f or non-character images , from being 
mistakenly recognized as character images and outputted as 
character code data by judging whether character images should 
be converted into character code data based on the degree 
of character continuity after character codes are recognized 
from character images. Moreover, it becomes possible to 
remove mistakenly recognized character code data without 
having to set up a severe threshold value in judging the 
certainty of recognizing character codes by means of using 
the degree of character continuity, so that it is possible 
to maintain a high rate of conversion of character images 
contained in the input image data into character code data* 
[0013] According to another aspect of the invention, it 
is a program product for image processing, said program product 
causing a computer to execute a process comprising the steps 



of : 1) recognizing character codes from character images in 
image data; 2) obtaining a degree of character continuity 
between a character image and neighboring character images 
thereof for any character image to which a character code 
has been recognized at said step 1) ; and 3) making a judgment 
on whether said character image should be converted into 
character code data based on said degree of character 
continuity. 

[0014] According to still another aspect of the invention, 
it is an image processing system, comprising a scanning device 
for scanning documents to obtain image data; and an image 
processing device comprising: a character recognition unit 
that recognizes character codes from character images in image 
data; a conversion unit for converting character images to 
character code data according to character codes; and a 
judgment unit that obtains a degree of character continuity, 
which is a degree of continuity between a character image 
and neighboring character images thereof, for any character 
image for which a character code has been recognized by said 
character recognition unit, and that makes a judgment on 
whether said character image should be converted into 
character code data based on said degree of character 
continuity. 

[0015] The objects, features, and characteristics of this 



invention other than those set forth above will become apparent 
from the description given herein below with reference to 
preferred embodiments illustrated in the accompanying 
drawings . 

BRIEF DESCRIPTIONS OF THE DRAWINGS 
[0016] Fig. 1 is a drawing that shows an example document 
wherein non-character graphics are embedded into character 
areas; 

[0017] Fig. 2 is a drawing that shows an example wherein 
character recognition process is applied to image data 
obtained by scanning the document of Fig. 1 and outputted 
by a conventional image recognition device; 
[0018] Fig. 3 is a drawing that shows a constitution of 
an information equipment system containing an image 
processing device according to an embodiment of the present 
invention; 

[0019] Fig. 4 is a block diagram that shows the general 
constitution of a computer shown in Fig. 3; 
[0020] Fig. 5 is a diagram for describing the position 
information of each character image; 

[0021] Fig. 6 is a flow chart that shows image processing; 
[ 0022 ] Fig . 7 A and Fig . 7B constitute a flow chart of output 
format judgment process shown in Fig. 6; 



[0023] Fig. 8 is a diagram for describing the judgment 
criteria for output formats; 

[0024] Fig. 9A through Fig. 9C are diagrams for describing 
character erasure process; 

[0025] Fig. 10 is a diagram of a case that lays out a 
combination of character code data, character image data, 
and image data consisting of graphic images, which are 
non-character images and thus are kept intact; 
[0026] Fig. 11 is a diagram that shows the contents of 
character code data, character image data, and graphic image 
data separately; and 

[0027] Fig. 12 is a diagram for describing another judgment 
criterion for output formats . 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
[0028] The embodiments of this invention will be described 
below with reference to the accompanying drawings . 
[0029] Fig. 3 is a drawing that shows a constitution of 
an information equipment system containing an 
image-processing device according to an embodiment of the 
present invention. 

[0030] An information equipment system 100 consists of 
information equipment such as a computer 10 capable of 
functioning as an image processing device, a scanner 50, or 



an image input device, and a printer 60, or an image output 
device, all of which are connected to a network N. 
[0031] The computer 10, the scanner 50, and the printer 
60 can mutually exchange data via the network N. 
[0032] Fig. 4 is a block diagram that shows the general 
constitution of the computer 10. The computer 10 has an 
interface (I/F) 18, which exchanges data with various 
information equipment; a scanner driver 12, which controls 
the scanner 50 ; a printer driver 14 , which controls the printer 
60; a ROM 15, which stores prescribed programs and data; a 
RAM 16, which temporarily stores data; an operating panel 
17, which is used by the user for viewing various information 
on displays and for inputting instructions; an image 
processing unit 13 (to be described in detail later), which 
performs various processes such as character recognition; 
and a main control unit 1 1 that controls various units mentioned 
above. The scanner driver 12 and the printer driver 14 are 
software and are stored in a memory device such as a hard 
disk (not shown). 

[0033] The operating instruction for the information 

equipment system shown in Fig. 3 can be made through the 
operating panel 17 of the computer 10. The operating panel 
17 in this embodiment is a virtual panel displayed on the 
display unit of the computer 10 and operating instructions 



are entered by means of operating such devices as the keyboard 
and the mouse, which are normally used for operating the 
computer 10. 

[0034] For example, the operating panel 17 shows a scan 
& file button and a scan & print button. When the scan & file 
button is operated to instruct an operation, the scanner 50 
will scan the document to produce image data, which will then 
be transmitted to the computer 10, image-processed by the 
image processing unit 13, and stored into the memory device 
of the computer 10 as a file of a specified format. 
[0035] When the scan & print button is operated to instruct 
an operation, the image data obtained by the scanner 50 will 
be image-processed by the image processing unit 13, and 
transmitted to the printer 60 to be printed on paper. 
[0036] Next, the image processing unit 13 will be described 
in detail in the following. 

[0037] The image processing unit 13 includes a character 
recognition unit 31, a character color detection unit 32, 
an output format judgment unit 33, a character image data 
generating unit 34, a character erasure unit 35, and a file 
generating unit 36. 

[0038] The character recognition unit 31 recognizes 

character codes from each character image to obtain character 
code data, and detects character recognition certainty , which 



is the degree of certainty of character code recognition. 
The method of recognizing a character code of a character 
image is done by matching the characteristic amount of each 
character image with a dictionary pattern which is stored 
in advance. The character recognition certainty is expressed 
by a scale of 10, where a larger value means a higher degree 
of recognition . The character recognition unit 3 1 recognizes 
the position information of a character image. The position 
information of a character image is obtained as the left top 
and right bottom coordinates of a rectangle that circumscribes 
the character image with the left top corner of the image 
data as the origin as shown in Fig. 5. The character 
recognition unit 31 further recognizes the attributes of a 
character such as the font type and size, the style of the 
character, i.e., bold or italic, underline, etc., from the 
character image . 

[0039] The font type can be recognized by classifying into 
groups such as the serif group and the sans serif group. The 
font size can be obtained by comparing the size of the 
circumscribing rectangle of each character image with the 
size of the character expressing the character code data using 
a normal font when the character code is recognized. The 
information such as the height and width of the character, 
i.e. , the font metric data, of specific representative fonts 



is stored in the memory as a data. 

[0040] The character color detection unit 32 detects the 
color of each character image, whose character code is 
recognized. 

[0041] The output format judgment unit 33 judges whether 
a character image, whose character code has been recognized, 
should be converted into character code data. In this 
embodiment, the output format judgment unit 33 checks all 
the character images, whose character codes have been 
recognized, and makes judgments individually whether it 
should convert them to character code data , or cutout character 
images from the input image data without converting them into 
character code data to generate character image data that 
are to be placed continuously without overlapping with 
adjacent character code data, or leave input image data intact 
in the original positions . Among the character images , whose 
character codes have been recognized, the character image 
data generating unit 34 cuts out the character images, for 
which judgments have been made to have character image data 
to be generated, from input image data. 

[0042] The character erasure unit 35 erases character 
images that have been converted to either character code data 
or character image data from the original image data, i.e., 
the input color image data. 



[0043] The file generating unit 36 generates a file of a 
specified format using character code data and character image 
data obtained from character images contained in the inputted 
image data. 

[0044] The operation of the image processing unit 13 will 
be described below referring to the flowcharts shown in Fig. 
6, Fig. 7A, and Fig. 7B. 

[0045] First, a conversion in the color space is made from 
the RGB color system to the Lab color system on the image 
data obtained by scanning a document by the scanner 50 (Sll) . 
The Lab color system is capable of handling data as common 
color information on the network N without any dependency 
on any specific information equipment such as scanners and 
printers . 

[0046] Next, the OCR preprocessing will be performed in 
order to form the image data that is to be inputted into the 
character recognition unit 31 (S12) . Because the character 
recognition unit 31 is going to recognize a character code 
by means of extracting attributes from the monochromatic 
binary image data in the later stage, the OCR preprocessing 
first binarizes the color image data represented by the Lab 
color system to generate monochromatic binary image data 
having character images in black and the background image 
in white. By detecting the L, a and b levels of the character 



images and the background image , the present embodiment 
performs the binarization process so that the character codes 
can be sufficiently recognized from the character images in 
the image data obtained by scanning the document even if the 
document contains color characters against a white background 
or black characters against a color background. In addition 
to this, the OCR preprocessing performs the removal of noises 
such as isolated points, the compensation for a case where 
the document is scanned in a skewed condition, the compensation 
for character images which are deformed or faint, etc. 
[ 0047 ] The monochromatic binary image data obtained by the 
OCR preprocessing is inputted into the character recognition 
unit 31 to have the character recognition processing to be 
performed on it (S13). The character recognition unit 31 
recognizes the character code of each character image in a 
character area after extracting character areas where 
character images exist from the monochromatic binary image 
data. The character recognition unit 31 is also capable of 
detecting the character recognition certainty, which is the 
degree of certainty for recognizing character codes, and 
further recognizes the character's attributes such as the 
font type and size, the style of the character, i.e., bold 
or italic, underline, etc., in addition to the position 
information of the character images. 



[0048] Next, the character color detection process will 
be conducted (S14). In other words, the character color 
detection unit 32 detects the color of each character image 
once the character code is recognized. Here, it uses the image 
data , which has been color converted into the Lab color system . 
The color detection of the character image is done by reading 
the values of L, a, and b in the character area, where the 
character image exists in the image data, and calculating 
the average values for the L, a, and b values for all the 
pixels that constitute the character image. The distinction 
between the pixels of the character image and the background 
image is done using the monochromatic binary image data 
obtained in the OCR preprocessing. Thus, it is possible to 
determine that black pixels in the monochromatic binary image 
data are the pixels of the character image and white pixels 
in the same are the pixels of the background image within 
the circumscribing rectangle of each character image, 
[0049] in case of outputting a character image after 
converting it into character code data, the character code 
data is outputted by specifying a color based on the color 
information of each character image detected by the character 
color detection unit 32. This will recreate the character 
color on the document. 

[0050] Next, the output format judgment process for 



individual character images within the character areas are 
performed (S15). The output format judgment unit 33 judges 
whether the character images, whose character codes have been 
recognized, should be converted into character code data. 
[0051] The output format judgment unit 33 makes its judgment 
on output format based on two basic judgment values . The first 
value for the judgment of the output format is the character 
recognition certainty, which is the degree of certainty for 
recognizing character codes. The second value for the 
judgment of the output format is the character continuity 
degree, which is the degree of continuity with neighboring 
characters, of a character image for which a character code 
has been recognized. The character continuity degree is a 
value obtained by evaluating the character-like 
characteristic from the continuity standpoint of the 
character attributes recognized from its character image. 
[0052] From the character images, whose character codes 
have been recognized, the output format judgment unit 33 
classifies the character images that are judged not to be 
converted into character code data into two output formats . 
If the degree of character recognition certainty is low for 
a given character image, although it is certain that it is 
a character image and even its character code has been 
recognized, the output format judgment unit 33 makes a judgment 



to cut out its character image from the input image data to 
generate a character image data that can be placed continuously 
with and not overlapping its neighboring character code data. 
If it is suspected that a character image, to which a character 
code has been assigned, has been recognized mistakenly and 
thus has been assigned character code despite the fact that 
it is a graphic image and non-character image, the output 
format judgment unit 33 makes a judgment to leave the character 
image intact in its original position in the input image data. 
[0053] The output format judgment process for character 
images, whose character codes have been recognized, will be 
described in detail in the following. 

[0054] the character recognition unit 31 outputs various 
information about individual character image that include 
character attributes, for example, the type of font such as 
the serif group and the sans serif group, font size, style 
such as bold and italic, underline, etc., the character 
recognition certainty, and position information of a 
character for which a character code has been recognized, 
in addition to the character code data. 

[0055] The character-likeness of a character image is 
evaluated by detecting the continuity of the character image 
concerning the position of the character image and the 
character attributes based on the above-mentioned information 



provided by the character recognition unit 31 . Although the 
system extracts character areas that have strong 
character-likeness from image data by checking position 
relations with character images in the surround area to a 
degree in the character recognition process , there can be 
a circumstance where an extracted character area includes 
non-character images or graphic images that are mistaken as 
character images. In order to avoid such isolated character 
images from being converted into character code data by mistake , 
the system of this embodiment makes a judgment on every 
character image whether the character image should be 
converted into character code data after the character code 
has been recognized for the particular character image. 
[0056] The character continuity degree, in other words, 
the evaluation amount of the character-likeness, is detected 
by checking five items , i.e., the distance between a character 
image whose character code has been recognized and its 
neighboring character images, the difference in font size 
between a character image whose character code has been 
recognized and its neighboring character images, the 
difference in font type between a character image whose 
character code has been recognized and its neighboring 
character images, the length of a continuous character image 
string in which a character image whose character code has 



been recognized is contained, and the difference in color 
between a character image whose character code has been 
recognized and its neighboring character images, and then 
calculating the evaluation point for continuity of said 
character image for each item mentioned above. 
[0057] More specifically, a variable "n" that represents 
the sequence number of each character image in question is 
initialized as shown in Fig. 7A (S21). 

[0058] The first evaluation point is calculated based on 
the distance between a character image whose character code 
has been recognized and its neighboring character images ( S22 ) . 
The position information (refer to Fig. 5) of ± k pieces of 
character images before and after the character image in 
question is determined and the first evaluation point Pl n is 
calculated based on the following formula. The suffix in the 
formula represents the sequence number of each character image 
(hereinafter the same). 



Pl n = Ml - min [\ {(Xln+1+x - X2 n+i ) + (Yl n+i+1 - Y2 n+i )}, 



wherein, if Pl n < 0 , Pl n = 0 ; Ml : the maximum value of the 
first evaluation point; XI: left X coordinate of the 
circumscribing rectangle of the character image whose 
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{(Xl n+i+1 - X2 n+i ) + (Yl n+i+1 - Y2 n+i )}] 



character code has been recognized; X2: right X coordinate 
of the circumscribing rectangle of the character image whose 
character code has been recognized; Yl: upper Y coordinate 
of the circumscribing rectangle of the character image whose 
character code has been recognized; and Y2 : lower Y coordinate 
of the circumscribing rectangle of the character image whose 
character code has been recognized. 

[0059] As can be seen from the above formula, the larger 
the distances between the character image in question and 
its neighboring character images is, the smaller the first 
evaluation point is . In other words , it shows that the further 
the character image in question is from its neighboring 
character images, the smaller the continuity is, and hence 
the lower its probability of being a character image is. 
[0060] It is also seen from the above formula that, in order 
to consider, for example, a character image located at the 
left end or right end of a row of multiple character images, 
the distance between the character image in question and its 
neighboring character image on its right as well as the distance 
between the character image in question and its neighboring 
character image on its left are determined, and the smaller 
of the two is used for calculation of the evaluation point. 
[0061] Next, the second evaluation point is calculated 
based on the difference in font size between a character image 



whose character code is recognized and its neighboring 
character images (S23). The font sizes of ± k pieces of 
character images before and after the character image in 
question are determined and the second evaluation point P2 n 

is calculated according to the following formula: 

k k 
P2 n = M2 - |S n+i - S„|) / {{^ S n+i ) / (2k+l)> 

wherein, if P2 n < 0, P2 n = 0; M2 : the maximum value of the 
second evaluation point; and S: font size. 
[0062] The denominator of the above formula represents the 
average font size of the neighboring character images. As 
can be seen from the above formula, the larger the difference 
in font size between the character image in question and its 
neighboring character images is, the smaller the continuity 
is, and hence the smaller the second evaluation point is. 
[ 0063 ] Next , the third evaluation point is calculated based 
on the difference in font type between the character image 
whose character code is recognized and its neighboring 
character images (S24). The font types of ± k pieces of 
character images before and after the character image in 
question are determined and the third evaluation point P3 n 
is calculated according to the following formula, where Nf 
is the number of character images whose font types are different 
from that of the character image in question: 



P3 n = M3 - Nf 

wherein, if P3 n < 0, P3 n =0; and M3 : the maximum value of the 
third evaluation point. 

[0064] As can be seen from the above formula, the larger 
the difference in font type between the character image in 
question and its neighboring character images is, the smaller 
the continuity is, and hence the smaller the third evaluation 
point is. 

[0065] Next, the fourth evaluation point is calculated 
based on the length of a string of continuous character images 
in which the character image whose character code is recognized 
is included (S25). The space between the adjacent 
circumscribing rectangles of individual character images are 
determined before and after the character image in question, 
and if a particular space is larger than a prescribed threshold 
value TH, it is judged as the edge of said character image 
string. It is preferable to vary the threshold value TH 
according to the average size of the neighboring character 
images . Assuming that the character image in question is the 
n-th character image and a group of character images from 
the il-th character image existing on the left side of the 
character image in question to the i2-th character image on 
the right side of the character image in question constitutes 
a single string of character images, the fourth evaluation 



n.l 



point P4 n is calculated according to the following formula: 

P4n = ±2 - il + 1 
wherein, if P4n > M4, P4n = M4; and M4: the maximum value 
of the fourth evaluation point. 

[0066] As can be seen from the above formula, the longer 
the character image string is, the larger the continuity is, 
and hence the larger the fourth evaluation point is. 
[0067] Lastly, the fifth evaluation point is calculated 
based on the difference in color between the character image 
whose character code has been recognized and its neighboring 
character images (S26) . The colors of ± k pieces of character 
images before and after the character image in question are 
determined and the fifth evaluation point P5 n is calculated 

according to the following formula: 

k k k 

P5 n = M5 - (J? |L n+i - L n | + l a *+i - a n| + ^ l b n+i " t>n | ) 

wherein, if P5 n < 0, P5 n =0; M5: the maximum value of the fifth 
evaluation point; and L, a, b: values expressed in character 
colors in the Lab color system* 

[0068] As can be seen from the above formula, the larger 
the difference in color between the character image in question 
and its neighboring character images is, the smaller the 
continuity is, and hence the smaller the fifth evaluation 
point is. 
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[0069] From the first through fifth evaluation points 
obtained as above, the degree of character continuity Pn as 
the general evaluation point can be calculated from the 
following formula (S27). 

Pn = Wl*Pl n + W2*P2 n + W3*P3 n + W4*P4 n + W5*P5 n 
wherein, Wl through W5 are the weights to be applied to each 
evaluation point and can be arbitrarily set. 
[0070] In the meanwhile, the degree of character 

recognition certainty Cn, which is the degree of certainty 
in recognizing the corresponding character code, is obtained 
for each character image in addition to the degree of character 
continuity Pn. 

[0071] The output format judgment unit 33 makes a judgment 
on the output format for each character image whose character 
code has been recognized based on the degree of character 
recognition certainty Cn and the degree of character 
continuity Pn, which is the degree of continuity with the 
neighboring character images . 

[0072] Fig. 8 is a diagram for describing the judgment 
criteria for the output format. 

[0073] First, two threshold values are set up as the 
threshold of the degree of character recognition certainty 
Cn, a smaller threshold CThl and a larger threshold CTh2, 
and also two threshold values are set up as the threshold 



of the degree of character continuity Pn, a smaller threshold 
PThl and a larger threshold PTh2. 

[0074] The output format judgment unit 33 first judges that 
a character image having a degree of character recognition 
certainty Cn larger than the larger threshold value CTh2 should 
be converted into character code data (S28: Yes; S33). The 
output format judgment unit 33 also judges that a character 
image having a degree of character recognition certainty Cn 
larger than the smaller threshold value CThl and a degree 
of character continuity Pn is larger than the smaller threshold 
value PThl should be converted into character code data on 
the judgment that the character code is correctly recognized 
(S29: Yes; S33). In other words, a character image having 
an extremely large degree of character recognition certainty 
(Cn > CTh2) is judged to be converted into character code 
data, and a character image having a medium degree of character 
recognition certainty (CThl < Cn < CTh2 ) is judged to be 
converted into character code data only when the degree of 
character continuity Pn is larger than PThl . 
[0075] Character images that are judged that they should 
not be converted into character code data at the 
above-mentioned two steps, S28andS29, are further classified 
into two output format categories based on their degrees of 
character continuity Pn. Specifically, of the character 



images that are judged not to be converted into character 
code data , those character images with the degrees of character 
continuity Pn larger than the threshold value PTh2 are judged 
to have their character image data generated by cutting out 
their character images from the input image data . On the other 
hand, of the character images that are judged not to be 
converted into character code data, those character images 
with the degrees of character continuity Pn smaller than the 
threshold value PTh2 are judged to be left intact in their 
original positions in the input image data. 
[0076] As can be seen from the above, the character images 
whose character codes are recognized by the character 
recognition unit 31 are finally classified into three 
categories by the output format judgment unit 33. 
Specifically, individual character images are divided into 
a category of character images, whose character codes are 
recognized beyond doubt, and which are consequently judged 
to be outputted as character code data (area Al in Fig. 8); 
a category of character images , which are judged to be character 
images beyond doubt but doubtful as to whether correct 
character code have been recognized, and which are 
consequently judged to be outputted as character image data 
that are arranged to be continuous to but not overlapping 
with neighboring character code data (A2 area in Fig. 8); 



and a category of character images, which have little 
continuity with neighboring character images , whose character 
codes have been mistakenly recognized despite the fact that 
they are non-character, graphic images, which are 
consequently judged to be left in the input image data intact 
at their original positions (A3 area in Fig. 8). 
[0077] After executing the above process, the variable "n" 
that represents the sequence number of the character image 
in question is incremented (S34) , and the process is repeated 
until all character images in the image data are treated (S3 5 ) . 
[0078] When the judgment process of the output format is 
completed on all character images in the image data , the program 
returns to the main flow chart shown in Fig. 6 to generate 
character image data (S16). The character image data 
generating unit 34 generates character image data by cutting 
out the image data within the circumscribing rectangle of 
each character image according to the position information 
of the character images outputted by the character recognition 
unit 31. 

[0079] The reason that the character image to be outputted 
as the character image data is cut out individually without 
leaving it in its original position as the image data obtained 
by scanning the document, is so that the character code data 
and the character image data do not overlap each other. In 



general, the input font and the output font seldom match with 
each other, so that the position where character code data 
converted from a character image is disposed will be slightly 
off the position of said character image within the input 
image data. As a result, if a character image is left as image 
data in its original position without cutting it out as a 
character image and character code data is disposed, it can 
cause a problem that the character code data overlaps with 
the character image in the original image data obtained by- 
scanning the document. Therefore, the present embodiment 
cuts out each character image to produce character image data 
and disposes the cutout character image data in succession 
to the place where the character code data is disposed with 
the output font. 

[0080] in generating character image data, the character 
image data generating unit 34 applies to image data within 
circumscribing rectangle of corresponding character image 
such processes as the edge enhancement and the gamma conversion 
applicable to said character image. This makes it possible 
to recreate even the character image data expressed in bit 
maps in sharp characters when printed on paper. 
[0081] Next, it performs the character erasure process 
( S 1 7 ) . After obtaining character code data or character image 
data from a character image, the character erasing unit 35 



erases the original character image, from which the character 
code data or the character image data has been obtained, from 
the color image data obtained by scanning the document. 
[0082] Fig. 9A through Fig. 9C are the drawings to describe 
the character erasure process. As shown in the drawings, a 
portion of the image data within the circumscribing rectangle 
of an individual character image (Fig. 9A) that corresponds 
to the character image is erased (Fig. 9B) , and is complimented 
by the surrounding image data (Fig. 9C) . The portion to be 
erased can be obtained by a process of expanding the black 
area of a monochromatic binary image data by two to three 
pixels in the circumscribing rectangle of said character image . 
The image data remaining after removing the portion that 
corresponds to the character image from the original image 
data obtained by scanning the document still holds image data 
other than the character image, i.e., images such as a 
photograph and a line drawing, or a background image as shown 
in Fig. 9B. 

[0083] By performing the abovementioned processes, as an 
example, character code data Dl and character image data D2 
are obtained from the character image within the image data, 
and image data D3 that is still holding graphic images that 
are non-character images are also obtained shown in Fig. 10. 
Fig. 11 is a diagram that shows the contents of character 



code data, character image data, and graphic image data 
separately. As shown in Fig. 10 and Fig. 11, non-character 
graphic images for which character codes are mistakenly 
recognized and exist as isolated character images are 
prevented from being converted into character code data and 
will not be outputted as mistakenly recognized character code 
data in the portion of the graphic image as shown in Fig. 
2. 

[0084] These character code data, character image data, 
and the image data are disposed in the page in a prescribed 
file format and stored (Si 8) . If the scan & print button is 
selected, the specified data will be transmitted to the printer 
60 and printed on paper. 

[0085] As can be seen from the above, after a character 
code has been recognized from a character image, a judgment 
is made on whether said character image should be converted 
into a character code data based on the degree of character 
continuity, so that this embodiment can efficiently prevent 
mistakenly recognizing a character code in a non-character 
graphic image and outputting character code data. 
[0086] Moreover, since it becomes possible to remove 
mistakenly recognized character code data not having to set 
up a higher threshold value for the degree of character 
continuity, it is possible to maintain a high ratio of 



conversion from character images in input image data to 
character code data. 

[0087 J It is obvious that this invention is not limited 
to the particular embodiments shown and described above but 
may be variously changed and modified without departing from 
the technical concept of this invention. 

[0088] For example, although it was described in the above 
embodiments to convert the color space of the input image 
data from the RGB color system to the Lab color system in 
the image processing, it is also possible to conduct the image 
processing using other color space such as the RGB color system 
of the image data obtained by scanning the document with the 
scanner 50. 

[0089] The method of calculating the first through fifth 
evaluation points addressing the continuity of character 
images is not limited to the formulas given above . For example , 
the second evaluation point P2n can be calculated with the 

following formula: 

k 

P2 n = M2 - | Sn-f-i+l - S n +i | 

[0090] Moreover, although the degree of character 

continuity Pn was determined considering the entire first 
through fifth evaluation points in the above embodiments , 
it is not limited to it. The degree of character continuity 



Pn can be determined based on only one or a combination of 
two or more points of the first through fifth evaluation points 
if so desired. 

[0091] Moreover, although it was described in the above 
embodiments that the output format judgment unit 33 makes 
a judgment on the output format concerning each character 
image based on the degree of character recognition certainty 
Cn and the degree of character continuity Pn according to 
;fi the method shown in Fig. 8, the invention is not limited to 

!Jf it. The output format judgment unit 33 can make a judgment 

on the output format on each character image based on a method 
of continuously varying the threshold value CTh of the degree 
pi of character recognition certainty Cn according to the degree 

go of character continuity Pn as shown in Fig. 12. 

N ; [0092] Moreover, if it is desired to use a simpler output 

format judgment, it is possible for the output format judgment 
unit 33 to make a judgment whether character images whose 
character codes have been recognized should be converted into 
character code data using the degree of character continuity 
detecting only whether the character codes have been 
recognized from the character images without detecting the 
degree of certainty of recognizing character codes from 
character images quantitatively (e.g., by a scale of 10). 
[0093] Moreover, although it was described in the above 
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embodiments using a computer as an example of the image 
processing device , the image processing device of the present 
invention is not limited to it* The invention can be applied 
to a scanner that performs the processes described above and 
transmits the result to a computer or a printer, a printer 
that receives image data directly from the scanner and performs 
the processes described above, or a color digital copying 
machine that performs all the processes from scanning of a 
document to printing on paper. 

[0094] The control of the image processing in the 

embodiments described above is performed by the main control 
unit 11 as it executes a specified program, which contains 
the processing sequence mentioned above (refer to Fig. 6, 
Fig. 7A and Fig. 7B), and this program can be provided also 
by means of computer readable recording medium (e.g. , floppy 
disk, CD-ROM) . The specified program can be provided either 
as an application software program, or can be built into 
information equipment such as a computer, digital copying 
machine, scanner and printer as a function of them, or the 
control server as a part of their functions. In the present 
invention, a program product includes a program itself and 
a computer readable recording medium that contains the 
program. 



